
Ensemble member selection using multi%2Dobjective optimization.

Graph Propositionalization for Random Forests.

Improving Fusion of Dimensionality Reduction Methods for Nearest Neighbor Classification.

Using uncertain chemical and thermal data to predict product quality in a casting process.

Extending Nearest Neighbor Classification with Spheres of Confidence.

Calibrating Random Forests.
A number of supervised learning methods have been introduced in the last decade. Unfortunately, the last comprehensive empirical evaluation of supervised learning was the Statlog Project in the early 90’s. We present a large-scale empirical comparison between ten supervised learning methods: SVMs, neural nets, logistic regression, naive bayes, memory-based learning, random forests, decision trees, bagged trees, boosted trees, and boosted stumps. We also examine the effect that calibrating the models via Platt Scaling and Isotonic Regression has on their performance. An important aspect of our study is the use of a variety of performance criteria to evaluate the learning methods. 1.
On the Use of Accuracy and Diversity Measures for Evaluating and Selecting Ensembles of Classifiers.

Comprehensible Models for Predicting Molecular Interaction with Heart%2DRegulating Genes.

The problem with ranking ensembles based on training or validation performance.
The main purpose of this study was to determine whether it is possible to somehow use results on training or validation data to estimate ensemble performance on novel data. With the specific setup evaluated; i.e. using ensembles built from a pool of independently trained neural networks and targeting diversity only implicitly, the answer is a resounding no. Experimentation, using 13 UCI datasets, shows that there is in general nothing to gain in performance on novel data by choosing an ensemble based on any of the training measures evaluated here. This is despite the fact that the measures evaluated include all the most frequently used; i.e. ensemble training and validation accuracy, base classifier training and validation accuracy, ensemble training and validation AUC and two diversity measures. The main reason is that all ensembles tend to have quite similar performance, unless we deliberately lower the accuracy of the base classifiers. The key consequence is, of course, that a data miner can do no better than picking an ensemble at random. In addition, the results indicate that it is futile to look for an algorithm aimed at optimizing ensemble performance by somehow selecting a subset of available base classifiers.
Chipper %2D A Novel Algorithm for Concept Description.





Using Background Knowledge for Graph Based Learning A Case Study in Chemoinformatics.

Maximizing the Area under the ROC Curve with Decision Lists and Rule Sets.

Learning to classify structured data by graph propositionalization.

Reducing High%2DDimensional Data by Principal Component Analysis vs. Random Projection for Nearest Neighbor Classification.

Resolving rule conflicts with double induction.

Resolving Rule Conflicts with Double Induction.

Classification with Intersecting Rules.
Several rule induction schemes generate hypotheses in the  form of unordered rule sets. One important problem that has to be addressed  when classifying examples with such hypotheses is how to deal  with overlapping rules that predict different classes. Previous approaches  to this problem calculate class probabilities based on the union of examples  covered by the overlapping rules (as in CN2) or assumes rule  independence (using naive Bayes). It is demonstrated that a significant  improvement in accuracy can be obtained if class probabilities are calculated  based on the intersection of the overlapping rules, or in case of  an empty intersection, based on as few intersecting regions as possible.
Rule Induction for Classification of Gene Expression Array Data.

Learning to recognize brain specific proteins based on low%2Dlevel features from on%2Dline prediction servers.

Learning to recognize brain specific proteins based on low%2Dlevel features from on%2Dline prediction servers.

Automatic Keyword Extraction Using Domain Knowledge.
Documents can be assigned keywords by frequency analysis  of the terms found in the document text, which arguably is the primary  source of knowledge about the document itself. By including a hierarchically  organised domain specific thesaurus as a second knowledge source  the quality of such keywords was improved considerably, as measured by  match to previously manually assigned keywords.
Classifying Uncovered Examples by Rule Stretching.
This paper is concerned with how to classify examples that  are not covered by any rule in an unordered hypothesis. Instead of assigning  the majority class to the uncovered examples, which is the standard  method, a novel method is presented that minimally generalises the  rules to include the uncovered examples. The new method, called Rule  Stretching, has been evaluated on several domains (using the inductive  logic programming system Virtual Predict for induction of the base hypothesis)  . The results show a signicant improvement over the standard  method.
Boosting interval based literals.
A supervised classification method for time series, even multivariate, is presented. It is based on boosting very simple classifiers: clauses with one literal in the body. The background predicates are based on temporal intervals. Two types of predicates are used: i) relative predicates, such as &#034;increases&#034; and &#034;stays&#034;, and ii) region predicates, such as &#034;always&#034; and &#034;sometime&#034;, which operate over regions in the domain of the variable. Experiments on different data sets, several of them obtained from the UCI ML and KDD repositories, show that the proposed method is highly competitive with previous approaches.
Learning First Order Logic Time Series Classifiers.
Departamento de Inform&#039;atica
Combining Divide%2Dand%2DConquer and Separate%2Dand%2DConquer for Efficient and Effective Rule Induction.
. Divide-and-Conquer (DAC) and Separate-and-Conquer (SAC)  are two strategies for rule induction that have been used extensively.  When searching for rules DAC is maximally conservative w.r.t. decisions  made during search for previous rules. This results in a very efficient  strategy, which however suffers from difficulties in effectively inducing  disjunctive concepts due to the replication problem. SAC on the other  hand is maximally liberal in the same respect. This allows for a larger  hypothesis space to be searched, which in many cases avoids the replication  problem but at the cost of lower efficiency. We present a hybrid  strategy called Reconsider-and-Conquer (RAC), which handles the replication  problem more effectively than DAC by reconsidering some of the  earlier decisions and allows for more efficient induction than SAC by  holding on to some of the decisions. We present experimental results  from propositional, numerical and relational domains demonstrating that  RAC si...
Learning First Order Logic Time Series Classifiers Rules and Boosting.
Departamento de Inform&#039;atica
Combining Divide%2Dand%2DConquer and Separate%2Dand%2DConquer for Efficient and Effective Rule Induction.
. Divide-and-Conquer (DAC) and Separate-and-Conquer (SAC)  are two strategies for rule induction that have been used extensively.  When searching for rules DAC is maximally conservative w.r.t. decisions  made during search for previous rules. This results in a very efficient  strategy, which however suffers from difficulties in effectively inducing  disjunctive concepts due to the replication problem. SAC on the other  hand is maximally liberal in the same respect. This allows for a larger  hypothesis space to be searched, which in many cases avoids the replication  problem but at the cost of lower efficiency. We present a hybrid  strategy called Reconsider-and-Conquer (RAC), which handles the replication  problem more effectively than DAC by reconsidering some of the  earlier decisions and allows for more efficient induction than SAC by  holding on to some of the decisions. We present experimental results  from propositional, numerical and relational domains demonstrating that  RAC si...
Induction of Logic Programs by Example%2DGuided Unfolding.
Resolution has been used as a specialisation operator in several approaches to top-down induction of logic programs. This operator allows the overly general hypothesis to be used as a declarative bias that restricts not only what predicate symbols can be used in produced hypotheses, but also how the predicates can be invoked. The two main strategies for topdown induction of logic programs, Covering and Divide-and-Conquer, are formalised using resolution as a specialisation operator, resulting in two strategies for performing example-guided unfolding. These strategies are compared both theoretically and experimentally. It is shown that the computational cost grows quadratically in the size of the example set for Covering, while it grows linearly for Divide-and-Conquer. This is also demonstrated by experiments, in which the amount ofwork performed by Covering is up to 30 times the amount ofwork performed by Divideand-Conquer. The theoretical analysis shows that the hypothesis space is larger for Covering, and thus more compact hypotheses may be found by this technique than by Divide-and-Conquer. However, it is shown that for each non-recursive hypothesis that can be produced by Covering, there is an equivalent hypothesis (w.r.t. the background predicates) that can be produced by Divide-and-Conquer. A major draw-back of Divide-and-Conquer, in contrast to Covering, is that it is not applicable to learning recursive de nitions. 1 1
Induction of Recursive Transfer Rules.
Transfer rules are used in bi-lingual translation systems for transferring a logical representation of a source language sentence into a logical representation of the corresponding target language sentence. This work studies induction of transfer rules from examples of corresponding pairs of source-target quasi logical formulae (QLFs). The main features of this problem are: i) more than one rule may needtobeproducedfromasingle example, ii) only positive examples are provided and iii) the produced hypothesis should be recursive. In an earlier study of this problem, a system was proposed in which hand-coded heuristics were employed for identifying non-recursive correspondences. In this work we study the case when non-recursive transfer rules have been given to the system instead of heuristics. Results from a preliminary experiment with English-French QLFs are presented, demonstrating that this information is su cient for the generation of generally applicable rules that can be used for transfer between previously unseen source and target QLFs. However, the experiment also shows that the system su ers from producing overly speci c rules, even when the problem of disallowing the derivation of other target QLFs than the correct one is not considered. Potential approaches to this problem are discussed. 1
Induction of Logic Programs by Example%2DGuided Unfolding.
Resolution has been used as a specialisation operator in several approaches to top-down induction of logic programs. This operator allows the overly general hypothesis to be used as a declarative bias that restricts not only what predicate symbols can be used in produced hypotheses, but also how the predicates can be invoked. The two main strategies for topdown induction of logic programs, Covering and Divide-and-Conquer, are formalised using resolution as a specialisation operator, resulting in two strategies for performing example-guided unfolding. These strategies are compared both theoretically and experimentally. It is shown that the computational cost grows quadratically in the size of the example set for Covering, while it grows linearly for Divide-and-Conquer. This is also demonstrated by experiments, in which the amount ofwork performed by Covering is up to 30 times the amount ofwork performed by Divideand-Conquer. The theoretical analysis shows that the hypothesis space is larger for Covering, and thus more compact hypotheses may be found by this technique than by Divide-and-Conquer. However, it is shown that for each non-recursive hypothesis that can be produced by Covering, there is an equivalent hypothesis (w.r.t. the background predicates) that can be produced by Divide-and-Conquer. A major draw-back of Divide-and-Conquer, in contrast to Covering, is that it is not applicable to learning recursive de nitions. 1 1
Predicate Invention and Learning from Positive Examples Only.
Abstract. Previous bias shift approaches to predicate invention are not applicable to learning from positive examples only, if a complete hypothesis can be found in the given language, as negative examples are required to determine whether new predicates should be invented or not. One approach to this problem is presented, MERLIN 2.0, which is a successor of a system in which predicate invention is guided by sequences of input clauses in SLD-refutations of positive and negative examples w.r.t. an overly general theory. In contrast to its predecessor which searches for the minimal nite-state automaton that can generate all positive and no negative sequences, MERLIN 2.0 uses a technique for inducing Hidden Markov Modelsfrom positive sequences only. This enables the system to invent new predicates without being triggered by negative examples. Another advantage of using this induction technique is that it allows for incremental learning. Experimental results are presented comparing MERLIN 2.0 with the positive only learning framework of Progol 4.2 and comparing the original induction technique with a new version that produces deterministic Hidden Markov Models. The results show that predicate invention may indeed be both necessary and possible when learning from positive examples only as well as it can be bene cial to keep the induced model deterministic. 1
IMPUT An Interactive Learning Tool Based on Program Specialization.

Integrating Algorithmic Debugging and Unfolding Transformation in an Interactive Learner.

Theory%2DGuideed Induction of Logic Programs by Inference of Regular Languages.

JIGSAW Puzzling together RUTH and SPECTRE %28Extended Abstract%29.

Specialization of Recursive Predicates.
semantics φ (the programmer’s intentions) where P does not satisfy φ, find out a new version P ’ of P such that P ’ satisfies φ. Approach: We proposed an approach for correcting overly general programs from positive examples by exploiting program synthesis techniques. The synthesized program, P’, is a specialization of the original one, P. In contrast to the previous approaches for logic program specialization, no negative examples were given as input but they will be discovered by the algorithm itself. The specialization process is performed according to the positive examples only. A method for refining logic programs into specialized version was then proposed. Results: The proposed approach was able to correct overly general programs using positive examples. We showed that positive examples can also be used for inducing finite-state machines, success sequences, that models the correct program. The failing sequences also exploited by theorem proved to produce counter-examples as in model checking, by composing substitutions used for inducing failing sequences. Conclusion: The contribution of the study was mainly the use of specification predicates to specialize an overly general logic program. Key words: Program specialization, theorem proving, positive/negative examples, folding/unfolding rules, finite-state machine
Covering vs. Divide%2Dand%2DConquer for Top%2DDown Induction of Logic Programs.
Covering and divide-and-conquer are two wellestablished  search techniques for top-down induction  of propositional theories. However,  for top-down induction of logic programs, only  covering has been formalized and used extensively.  In this work, the divide-and-conquer  technique is formalized as well and compared  to the covering technique in a logic programming  framework. Covering works by repeatedly  specializing an overly general hypothesis,  on each iteration focusing on finding a  clause with a high coverage of positive examples.  Divide-and-conquer works by specializing  an overly general hypothesis once, focusing  on discriminating positive from negative  examples. Experimental results are presented  demonstrating that there are cases when more  accurate hypotheses can be found by divideand  -conquer than by covering. Moreover, since  covering considers the same alternatives repeatedly  it tends to be less efficient than divideand  -conquer, which never considers the same  ...
Improving Example%2DGuided Unfolding.

Optimizing Horn Clause Logic Programs for Particular Modes of Use An Analysis of Explanation%2DBased Learning and Partial Evaluation.

Generalizing the Order of Goals as an Approach to Generalizing Number.

